Function Approximation for Continuous Constrained MDPs

In this work we apply function approximation techniques to solve continuous, constrained Markov Decision Processes (MDPs). Many real-world robot planning problems are best represented as MDPs with a continuous state space. However, in many scenarios constraints must be accounted for as well. These constraints are treated probabilistically, with a bound on the constraint violation probability. Existing function approximation literature does not account for constraints, and existing constrained MDP work assumes a discrete state space. We seek to bridge this gap by maintaining two continuous value functions one for the reward and one for the constraint violation probability and approximate these value functions by writing them as a linear combination of a set of basis functions (not necessarily the same set for both value functions). We present some simulation results and investigate the impact of choice of basis functions on the quality of policies generated.